Generalized Shannon Code Minimizes the Maximal Redundancy

Source coding, also known as data compression, is an area of information theory that deals with the design and performance evaluation of optimal codes for data compression. In 1952 Hu man constructed his optimal code that minimizes the average code length among all pre x codes for known sources. Actually, Hu man codes minimizes the average redundancy de ned as the di erence between the code length and the entropy of the source. Interestingly enough, no optimal code is known for other popular optimization criterion such as the maximal redundancy de ned as the maximum of the pointwise redundancy over all source sequences. We rst prove that a generalized Shannon code minimizes the maximal redundancy among all pre x codes, and present an eÆcient implementation of the optimal code. Then we compute precisely its redundancy for memoryless sources. Finally, we study universal codes for unknown source distributions. We adopt the minimax approach and search for the best code for the worst source. We establish that such redundancy is a sum of the likelihood estimator and the redundancy of the generalize code computed for the maximum likelihood distribution. This replaces Shtarkov's bound by an exact formula. We also compute precisely the maximal minimax for a class of memoryless sources. The main ndings of this paper are established by techniques that belong to the toolkit of the \analytic analysis of algorithms" such as theory of distribution of sequences modulo 1 and Fourier series. These methods have already found applications in other problems of information theory, and they constitute the so called analytic information theory.